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Navigation System Testbed 

David Chelmins, O. Scott Sands, and Aaron Swa nk 
National Aeronautics and Space Administration 
Glenn Research Center 
Cleveland, Ohio 44135 

Abstract 

NASA is interested in finding new methods of surface navigation to allow astronauts to navigate on 
the lunar surface. In support of the Vision for Space Exploration, the NASA Glenn Research Center 
developed the Lunar Extra- Vehicular Activity Crewmember Location Determination System and 
performed testing at the Desert Research and Technology Studies event in 2009. A significant amount of 
sensor data was recorded during nine tests performed with six test subjects. This paper provides the 
procedure, formulas, and techniques for data analysis, as well as commentary on applications. 

1.0 Introduction 

The National Aeronautics and Space Administration (NASA) Vision for Space Exploration, 
announced by President George W. Bush in 2004, calls for the safe return of astronauts to the Moon. In 
support of the Vision, NASA is considering methods of surface navigation for crewmembers performing 
extra-vehicular activity (EVA). 

The Digital Communications and Navigation Branch at the NASA Glenn Research Center (GRC) has 
developed a mobile navigation testbed, the Lunar Extra- Vehicular Activity Crewmember Location 
Determination System (LECLDS), to evaluate techniques for human exploration. LECLDS consists of a 
Global Positioning System (GPS) receiver, an inertial measurement unit (IMU), a digital compass, a 
navigation processor, and a handheld display unit. LECLDS was demonstrated at the 2009 NASA Desert 
Research and Technology Studies (D-RATS) event held at the Black Point Lava Flow near Flagstaff, 
Arizona. 

During the D-RATS demonstration, test subjects used the navigation testbed to walk between pre- 
determined waypoints. Data was collected to gauge the test subject’s ability to navigate (Measures of 
Effectiveness, MoEs) as a function of a level of performance (Measures of Performance, MoPs) of the 
LECLDS testbed. Navigation data was displayed in real-time to the test subject, and raw measurements 
were recorded simultaneously for post-processing. 

This report covers the steps taken to analyze data from D-RATS 2009. This includes a brief overview 
of the log formats and time alignment strategy used for LECLDS, as well as the manual techniques 
needed to reduce the data set to useful segments. A number of data analysis metrics are presented, with 
commentary on applications. Finally, recommendations are made for ways to improve the collected data 
to make the analysis more reliable. 

2.0 LECLDS 2009 Logs and Data Formats 

2.1 Time Alignment 

LECLDS stored data in a number of different ASCII formats, with several time alignment strategies. 
Log files were recorded for the ground truth digital compass, the GPS interface program, the GPS 
information from the MATLAB navigation filter, the IMU information from the navigation filter, and the 
navigation filter position determination information. These logs are summarized in Table I. 
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TABLE I.— LECLDS DATA LOGS 


Type of data 

Example filename 

Data 

delimiter 

Time alignment 

Digital Compass 

COMPASSLOG090 1 09_0453 .csv 

Comma 

System timestamp (ns) 

GPS Interface 

gps.log.txt 

Tab 

GPS time (ms) 

System timestamp (sec) 
Processor ticks (ticks/sec) 

Filter GPS Log 

GPSLOG_090409_0551.csv 

Comma 

GPS time (sec) 

System timestamp (ns) 

Filter IMU Log 

IMULOG 090409_05 5 1 .csv 

Comma 

System timestamp (ns) 

Filter Log 

NAVLOG 090409 05 5 1 .csv 

Comma 

GPS time (sec) 

System timestamp (ns) 


During post-processing, time alignment between the data logs was conducted in a Microsoft Excel 
spreadsheet for each test run. The navigation filter GPS log was used as the authoritative source of time 
alignment, since it linked GPS time with the system timestamp. Next, the Filter IMU Log was imported 
as new columns in the same file. The difference between the IMU system timestamp and the GPS system 
timestamp was computed, and the IMU log file information was adjusted up and down the spreadsheet 
rows until the time error between the two was close to zero. In general, once the start of the two logs was 
synchronized, the time error between them did not grow. In the event that one of the logs skipped time 
(for example, the IMU recorded a double value), adjustments were made to correct the skip by deleting or 
adding a bla nk line. 

After synchronizing the filter GPS log and IMU log, the navigation filter log was loaded into the 
same spreadsheet in a similar fashion. The filter log was then synchronized to the filter GPS log by 
comparing the GPS times. There was very little deviation between these logs since they both are derived 
from the same source of data; in most cases, no adjustment had to be made. 

The GPS interface log was not synchronized with the other logs, since the Filter GPS Log already 
contained a subset of the data from the interface log. However, in the case of extreme data corruption, the 
GPS interface log was manipulated into the Filter GPS Log format and used in place of the Filter GPS 
Log. This is acceptable because the subset of data in question was equivalent between both logs. 

2.2 Multiple Log Files 

Frequently a single test would have many different log files associated with it. Every time the 
navigation processor was stopped, the Digital Compass, Filter GPS, Filter IMU, and Filter logs were re- 
created using a new filename, and the GPS interface log was appended. If this occurred during a test, it 
meant that the test data was spread across many instances of the same log. 

In most cases, the additional logs were appended as rows to the existing spreadsheet. There usually would 
be a large jump in time at the point where the log was appended; this was documented but permitted, 
since it was more important to have synchronization across all of the logs in Table 1. 

2.3 Visualizing the Logs 

Using the new, composite spreadsheet containing time-synchronized data, it was possible to generate 
KML files that allowed the walkback tests to be visualized in Google Earth. In order to make the 
conversion, formulas were used to convert the Cartesian position fixes for GPS and the filter into 
longitude, latitude, and elevation. 

The latitude, longitude, and elevation formulas follow the World Geodetic System 84 (WGS84) 
model for Cartesian coordinates x, y, and z. Constants a and g are defined for Earth, yielding the model: 

a := 6378137 
g := 0.081819190842622 


N AS A/TM— 20 11-217113 


2 




cos 


( 


Latitude :=tan 


y/x 2 + y 2 — g 2 a l cos [ tan 


^ajl- g 2 yjx 2 + y 2 \\ \ 


az 


Z+ 7? T 


g 2 a 


g z 


sin tan 


ciyjl - g 2 yjx 2 + y‘ 


az 


Elevation := 


Longitude := tan 1 | — 
yjx 2 + y 2 


cos (Latitude) ^/l — ^2 sin ( Latitude ) 


Google Earth KML files can record data as individual points or “linestrings”, which consist of a line 
that is drawn to connect all of the points. The linestring form of the KML file is more useful for 
visualizing the ground truth position fixes, since the ground truth does not have large jumps. The 
navigation filter position fixes are better visualized using individual points. 

Additional comma-separated value (CSV) files were created from field notes to represent the 
recorded positions of the waypoints, antenna tower, and base camp. The positions were determined using 
a handheld GPS receiver, converted to KML, and loaded into Google Earth using distinctive icons. 


3.0 Data Modifications Prior to Analysis 

3.1 Determining Starting and Ending Locations 

The testbed began recording position fixes as soon as it was activated, and it continued to log data 
until it was stopped. This made analysis difficult because there was an unknown amount of time at the 
beginning and end of each test that did not contain useful data. These amounts of time were removed by 
examining the ground truth position fix points in Google Earth. 

The starting point in Figure 1, for example, must be determined visually. The walkback test in this 
case started somewhere within the large cluster of points on the left, and proceeded to the right. The 
actual starting time is determined visually as the first point outside of the cluster that is in sequence with 
the trail to the right. In this case, the point 40983 1 would be chosen as the starting point for the test. The 
label 40983 1 is the GPS time of week, so this value also represents the starting time. 

A similar point selection method was applied to determine the finishing time and location. The 
finishing location is the last point in the sequence that is still outside of the cluster of points near the end 
of the walk. 

Once the starting and finishing times were determined, all points outside of these times were saved in 
a separate, isolated part of the composite spreadsheet. Only times between the start and finish were 
considered for data analysis. 
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Figure 1 . — Starting point surrounded by a cluster of position fixes 
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3.2 Determining Waypoint Arrival and Departure Times 

In order to generate a useful data set, it was necessary to consider the individual walk segments 
between two waypoints. At first the entire test was analyzed as a single walk, however this was not 
accurate because the errors experienced by the walkback team were not uniform across the test. Based on 
the geometry of the radiometric nodes, certain waypoint intervals experienced larger position errors than 
other intervals. Averaging over the entire run did not capture the location-dependent effect of these errors. 

The arrival and departure times for individual waypoints were determined visually, similar to the 
beginning and ending times, although with less precision. For example, the waypoint in Figure 2 has a 
significant amount of ambiguity. The arrival time was determined to be 410445, and the departure time 
could be chosen as 410528. 

The difficulty in choosing arrival and departure times visually is that the choice tends to make the 
navigation system appear to perform better. It is possible that the team started walking at 4 1 047 1 and was 
not sure which way to go, resulting in the clusters around 410491 and 410510. It is also possible that the 
team decided to take a break and just walked around the area for a minute, in which case the choice of 
starting location would be correct. 

Another issue is that the waypoint location is subject to interpretation. In Figure 2, none of the ground 
truth points reached the manually-determined waypoint. Future tests must have better accuracy on the 
waypoint locations, or have more explicit ways of determining arrival and departure times and locations. 

3.3 Determining Break Times 

The walkback team was instructed to take periodic breaks, which were occasionally not recorded in 
the test log when the team did not communicate that they were planning to take a break. This presents an 
analysis difficulty because it was hard to determine whether the team was standing still and trying to 
figure out what direction to walk, or whether they were standing still while taking a break. 

Many breaks were taken at waypoints, so the procedure for determining waypoint arrival and 
departure often took precedent. For breaks not taken at waypoints, the analysis procedure relied entirely 
on the hand-recorded test director’s log for guidance. Even if there was a cluster of points that could be a 
break, this data was not removed unless the test log indicated the team took a break. This distinction is 
important because it preserves the “human factors” components (confusion and abrupt direction changes) 
of the test, as seen in Figure 3. 
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Figure 2. — Waypoint arrival and departure ambiguity 
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Figure 3. — Visual inspection ambiguity in determining 
breaks 


The times between the handwritten test log and the GPS clock tend to vary from 1 to 3 min across the 
tests, so there is additional ambiguity when designating a portion of the data a “break”. In general, when 
the test log indicates the team took a break, the data is scanned for a large cluster of points around the 
time recorded in the test log. When these points are found, the arrival and departure time is determined 
and the points between the two times are removed from the analysis data set. 

3.4 Determining System Errors and Data Corruption 

Every so often the system needed to be rebooted or the software needed to be restarted to correct 
errors. When this occurred, multiple logs were created for the same test. System errors also affected the 
test subject’s interpretation of the data. Sometimes the position display did not update, and the compass 
remained fixed on the same heading when a system error occurred. It was not immediately clear to that 
the system had stopped, so the test subject continued walking on the same heading. 

System errors were easy to detect in the data set because they were seen as large jumps in time or 
position. However, they presented an interesting problem from an analysis perspective. In general, the 
walkback team was walking from point A to C. The system error occurred at point Bl, somewhere 
between A and C. The walkback team discovered the system error at point B2, which was after Bl but 
before arriving at C. The gap between Bl and B2 is shown in Figure 4. 

When a system error occurred, the walk between two waypoints was split into two segments. The first 
segment covered the distance from A to Bl. This segment was discarded because the team did not reach 
the intended waypoint C, so it was impossible to draw conclusions about how effectively they navigated 
to point Bl when their target was point C. The second segment, B2 to C, was analyzed by considering B2 
as the starting location and C as the destination. The effectiveness of the system was then considered only 
over the interval B2 to C. 
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Figure 4. — Typical gap associated with a system error 


3.5 Reconstructing the Dataset by Segments 

After the entire walk was split into waypoints, breaks, and system errors, it was possible to 
reconstruct the dataset into individual segments. A single segment is the time and location of departure 
from the starting waypoint to arrival at the destination waypoint, excluding any breaks or errors between 
the two waypoints. The composite data set then was modified so that there were several empty rows 
between segments. 

3.6 Adding Waypoint Information to the Dataset 

After rearranging the dataset into segments, the position fixes for each of the individual segments 
were tagged according to the source and destination waypoints. This was done using the Google Earth 
map, as well as reading the test log and the pre-determined waypoint sequence. 

The previous waypoint’s latitude and longitude were determined using two methods. In the first 
method, the latitude and longitude coordinates from the field notes were loaded into the spreadsheet. In 
the second method, the latitude and longitude coordinates of the last ground truth point in the previous 
segment were used. That is, when walking on the path between A->B->C waypoints, the B->C portion 
considers the stopping point of the A->B portion as its starting point. This was done to provide two 
methods of calculating ideal path, since the walking path tended to start/end near a waypoint but usually 
not on a waypoint, as in Figure 5. 
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Figure 5. — Mismatch of waypoint and path locations 

The destination waypoint’s latitude and longitude were taken directly from the field notes, because a 
similar ambiguity did not exist for the destination. 


4.0 Data Analysis 

4.1 Idealized Waypoint to Waypoint Traverse Analysis 

The ideal two-dimensional path between two waypoints is a straight line following a single heading. 
Therefore, the optimal traverse path can be described by a distance and heading calculated between the 
source and destination waypoints. For this analysis, the WGS84 model for curvature is not followed since 
the waypoints are close enough together that the error is minimal (<1 m in most cases). 

To simplify variable names, a new naming convention is introduced for the following sections. In 
general, capital letters are used to indicate ground truth position, whereas lowercase letters are used to 
indicate position recorded by hand (e.g., from field notes) or determined from the filter position fixes. The 
prefix of the variable name indicates the data type being expressed. Examples are provided: 


h _pG 

h _pn 

a _pGpn 

d_PG 

la _p 
lo _p 
la_G 

sj 

xj 

vxj 


Heading from previous waypoint (p ) position based on field notes to current 
ground truth position (G) 

Heading from previous waypoint (p ) position based on field notes to next 
waypoint (n) based on field notes 

Positive angle between the headings of (previous waypoint <p> to current 
ground truth position) and (previous waypoint <p> to next waypoint <ri>) 
Distance from previous waypoint ( P ) closest ground truth position to current 
ground truth position (G) 

Latitude of the previous waypoint (p ) position based on field notes 
Longitude of the previous waypoint (p ) position based on field notes 
Latitude of the current ground truth position (G) 

Speed of the filter (/) determined from filter velocity vectors 
X-coordinate for the filter position fix (/) 

X-coordinate of the filter velocity vector (/) 


The ideal traverse heading function h _pn calculates heading assuming a linear latitude and longitude 
scale. The longitude is corrected according to the average of the source and destination latitudes. This 
function is defined in degrees as: 
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The ideal traverse heading function operates on small, 90° arcs to avoid ambiguity across platforms 
and programming languages. There is no correction for crossing the equator or prime meridian, but 
luckily the test subjects did not walk that far off course! Effectively, this translates into a clockwise angle 
starting on the positive Y-axis, as shown in Figure 6. 

The ideal traverse distance function is defined using the spherical law of cosines (Ref. 1). This 
equation is computationally simple and provides reasonable accuracy to 1 m. The function is defined: 


/4UU4147UN , r /■ \ N 

cL_pn •■= I J cos 1 [cos(/a p J cos \lo p ) cos (Za n ) cos (lo n ) 

+ cos(/a p ) sin(ZOp) cos(/a n ) sin(Zo n ) + sin(/a p ) sin(/a n )] 


The initial constant quantity represents the radius of the Earth based on the mean circumference. 
Using trigonometric identities, the equation can be simplified to the more familiar: 

/40041470\ r , . , . , . 

cL_pn ■= I I cos 1 [cos(/a p J cos (Za n ) cos [lo n — lo v ) + sin(Zap) sin(/a n )J 
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4.2 Path Conformance Analysis 

One measure of effectiveness of the navigation filter is the ability of the walkback team to adhere to 
the ideal traverse heading, which would minimize the walking distance. The path conformance can be 
viewed in terms of the positive 90° distance from the ideal traverse path for each position fix. In other 
words, path conformance is the distance back to the ideal traverse path, as shown in Figure 7. 

Recognizing that the 90° deviation forms a right triangle between the starting location, the current 
location, and the ideal location, the path conformance can be calculated. The right triangle shown in red 
(Figure 7) illustrates this concept. 

The 90° deviation is calculated from several values. First, a distance is determined between the 
starting location (i.e., the ground truth location where the previous walk segment ended) and the current 
ground truth location. The angle between the ideal path and the actual path is also calculated in terms of a 
reference axis as visualized in Figure 6. Finally, the deviation is calculated using a right-triangle formula. 
These measurements are illustrated in Figure 8, where h _pG is the heading from the previous waypoint 
position to the current ground truth position, h _pn is the ideal heading from the previous waypoint to the 
next waypoint, a _pGpn is the angle between h _pG and h _pn, and d_PG is the distance from the previous 
waypoint’s ground truth position to the current ground truth position. 



d PG 
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The formulas for the corresponding 90° deviation quantities in degrees are: 
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The 90° path deviance is then determined to be: 

deviance ■■= d PG * sin (a pGpn ) 

This analysis is performed in 2-D primarily to reduce computational complexity. However, it is 
unlikely that a 3-D measurement would vary greatly. Since the testbed focuses on surface navigation, 
LECLDS uses a digital elevation map to tie each filter position fix to the Earth’s surface. Even if this 
were not the case, the test subjects still are restricted to ground traverses. The curvature of the earth is 
insignificant over a typical 1 -km test segment. 


4,3 Walking Path Headings 

Several computations can be made to determine how well the test subjects were able to use the 
navigation system to navigate to a waypoint. Many of these computations, such as walking time or speed, 
are not compared easily between test subjects of varying physical capabilities. However, in general, all 
test subjects should be able to use the system to generally walk in the correct direction. 

The first equation, h _pG, was calculated for the path conformance analysis. The heading h _pG is 
determined from previous waypoint position, as determined from field notes, to the current ground truth 
position. Vectors are drawn for every second of the traverse, and a new heading is computed, as shown in 
Figure 9. 

Since the ground truth position changes during each second of the test, h _pG is recomputed during 
the entire walk as the heading from the previous waypoint to the new ground truth position. As the 
distance between the two points becomes larger, the heading tends to smooth out. Eventually, h _pG will 
converge to the ideal waypoint-to-waypoint heading when the team arrives at the destination waypoint. 
This concept is applied to a walk segment in Figure 10. 

In some cases, it may be better to use h_PG, which is the heading between the final ground truth 
position of the last segment of the walk and the current ground truth position. This metric is more useful 
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when the walkback team does not actually approach the waypoint position recorded in the field notes, 
such as in Figure 10. In general, this only affects the first minute of the walk significantly because the 
angles tend to converge once the distance is sufficient. Determination of h_PG is shown in Figure 1 1 . 



Figure 9. — h _pG heading computation 



Figure 10. — Example of heading from previous waypoint to current ground truth position 



N AS A/TM— 20 11-217113 


11 




In the ideal case, the walkback team should begin and end each segment of the walk at a waypoint. 
The waypoint position should match the position in the field notes, which would lead to the two headings 
h _pG and h_PG being equal. Otherwise, it is difficult to decide which metric to use since they vary 
subjectively. In the situation where the segment does not begin at a waypoint (e.g., after restarting the 
system following an error), the system starting coordinates are used for la _p, lo _p, la_P, and lo_P, and 
therefore h _pG equals h_PG. 

The equation for h_PG in degrees is given as: 
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Another way of considering heading is to examine h_Gn, which provides the ideal heading from the 
current ground truth position to the next waypoint (based on field notes). When the test subject walks off 
of the ideal waypoint-to-waypoint course, it no longer makes sense to use the ideal waypoint-to-waypoint 
heading as a baseline. The new baseline heading is the one that delivers the shortest path from the current 
location to the next waypoint, or h_Gn. This is illustrated in Figure 12. 

The formula for h_Gn in degrees is given: 
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(Zo G - lo n ) cos(- 
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Id 
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(lo n - lo G ) cos ^ a n + la c j 
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Figure 13. — h_G course computation 


The testbed incorporated a digital compass, which provided instantaneous heading. The sampled 
compass value will be called h_D. Since the digital compass data is synchronized with the GPS and 
navigation filter data, the raw headings can be used to compare to the baselines like h _pG, h_PG , and 
h_Gn. However, the digital compass can suffer from alignment issues and electromagnetic interference, 
making a direct comparison difficult between the digital compass heading and position-based courses. 

Course, or heading derived from position change, can be used to determine what direction the test 
subject was walking over a 1-sec interval. This is illustrated in Figure 13. The computation is performed 
using subsequent ground truth position fixes and the formula for h_G in degrees: 


tan' 


-! O Gfc-^Gfc-l) C0S Oc fe ) 


O a Gfc la G k - 1) 


for la Gk > la G fc _ 1 and lo Gk > /o Gfe _ 1 


h G {k ) := 


270 + tan " -lo 0 Jcos(la 0 J for ,ac ‘ ^ and K ‘ 


180 + tan 


_! 0°Gfc_! — l°G k ) COs(Za Gfe ) 


lCL G k - ! la G k 


for la Gk < la Gk _ 1 and Zo Gfe < /o Gfe _ 1 


(1 (Xq, IcLgis) 

90 + tan f0r la “ K la - l0s ^ 


Finally, the filter heading displayed to the test subject is also available for analysis. In general, the 
filter heading is computed the same as h_G, except with filter position fixes, however the filter can use 
various smoothing techniques that may affect the display. The filter heading, on a second-by-second 
basis, is called h J. 


4.4 Walking Path Heading Errors 

Evaluation of the test subject’s heading can be done according to several baselines, as mentioned in 
the previous section. The first technique compares h _pG to h _pn to determine how well the test subject’s 
heading converges to the ideal point-to-point heading. This technique is best used when the test subject 
begins the walk on a waypoint. The corresponding equation, a _pGpn, was used for the path conformance 
metric. 

When the test does not begin on a waypoint, but close to a waypoint, it is useful to examine heading 
error between the previous waypoint’s ground truth position and the ideal heading. This is often a difficult 
comparison to make when the distance between the field position and the actual starting position are 
large; in this case, the shortest vector to the next waypoint is no longer h _pn and other techniques should 
be used. This equation, a PGpn, is given as: 
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_ ( \h-PG hpn | 360 for (h PG h pn ) > 180 

O-PGpn * — ) 

l \hp G - h m \ for i h PG - hpn) ^ 180 

It is possible to compare the ground truth headings obtained from the digital compass with the ideal 
traverse heading between waypoints. This comparison should show whether the test subject was pointing 
in the right direction. In general, the analysis is useful in a qualitative sense, however it might not be as 
useful for determining filter effectiveness since there can be errors (such as electromagnetic interference 
from other testbed electronics) that would not cancel when compared to consecutive position fixes. The 
equation, a_Dpn, is given as: 

_ ( | h D - h vn | - 360 for (h D - h pn ) > 180 

a Dpn ) . . , s 

{ \ h D - hp n | for ( h D - hpn ) < 180 

The test subjects were given second-by-second feedback on their current walking heading, as 
determined using a course computation over subsequent filter position fixes. This heading can be 
compared versus the ideal heading (which was also displayed on the screen). However, it is important to 
note that the ideal heading here is the heading between the two waypoints, which is no longer ideal as 
soon as the test subject walks away from the point-to-point line between the two waypoints. This 
equation, a Jpn , is given as: 

_( \hf-h pn \ -360 for (h f - h pn ) > 180 

d'f'DYl * ) 

( \hf ~ hpn\ for (h f - hpn) < 180 

One of the more useful metrics is to examine error between the filter heading displayed to the test 
subject and the ideal heading from the current ground truth location to the next waypoint. This is 
interesting because the baseline (ideal) heading changes on a second-by-second basis as the test subject 
navigates. The ideal course will always be the straight line between the current location and the next 
waypoint. This equation, a _fGn, is given as: 

_ ( \h f ~h Gn \ - 360 for (hf — h Gn ) >180 

d'f Gyl * — ] 

( | h f - h Gn | for (h f - h Gn ) < 180 

The accuracy of the filter heading displayed to the user can be compared to the digital compass 
ground truth heading. Again, this comparison may contain errors that are not cancelled out since the 
magnetic compass does not have correlated errors with the GPS ground truth. Still, this is a useful metric 
in observing the general “correctness” of the filter heading being shown to the test subject. Significant 
errors here would point to the need for a heading sensor (e.g., sun sensor) as opposed to the radiometric 
course currently used in LECLDS. This equation, a JD, is given as: 

| hf — h D | — 360 for (hf — h D ) > 180 
| hf — h D | for (hf — h D ) < 180 

It is also possible to compare the digital compass heading with the ideal heading to the next waypoint. 
The advantage of this comparison is that it allows analysis to occur on the actual direction the test subject 
was pointing during the walkback. Other methods that only use position changes indicate the direction the 
test subject was moving, which does not indicate if the test subject was looking around for the waypoint. 
This equation, a_DGn , is given as: 
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, = [| I h G n -h D | - 360 1 for (h Gn - h D ) > 180 
DGU ' 1 \h Gn -h D \for(h Gn -h D )<180 

The filter heading can be compared with the ground truth course (using consecutive, 1-sec position 
fixes). This technique is similar to the a JD calculation, except common errors should be cancelled since 
both values are determined indirectly from GPS data. The calculation allows the filter heading 
performance to be examined. This equation, a JG , is given as: 


a fc 


(j \hf — h G \ - 360 1 for (h f - h G ) > 180 

) I . , I ~ . 


The ground truth course also can be compared to the ideal waypoint-to-waypoint heading. This is a 
useful metric for the beginning of a walk, or as long as h _pn is still a good baseline. The equation for 
a_Gpn is given as: 


a, 


Gpn " 



360 


for (h G - h pn ) > 180 


for (h G - h pn ) < 180 


4.5 Filter Solution Analysis 

One principle behind LECLDS testing is that the accuracy of the data presented to a test subject is a 
function of the filter being used for the walkback test. This can be characterized in terms of the 
computation error (versus some ground truth quantity) or derivative characteristics (versus ground truth or 
some derivative of ground truth). Equations that judge the performance of the filter, independent of the 
walkback, are considered to be Measurements of Performance (MoPs). 

The distance between two filter position estimates, d _fkfk-\, can be used to show the variance of 
position fixes around a certain point of time. The sum of all of these values during a test shows the total 
path length from the perspective of the navigation filter. These values are useful in showing a qualitative 
comparison of filters, since large variances will result in longer filter paths (versus the ground truth path). 
However, the individual distance between points is not a useful metric, otherwise, since it is only 
mathematical error; the test subject will never walk the distance indicated by the filter as long as some 
filter error exists. The corresponding equations are given as: 


d f(k)f(k- 1) 


J( x /ck) - x f(k- d) + (y/oc) - ym-i)) + 0 


■/(fc) 




d f{k)f{k- 1 ) 


The positive elevation of the filter also is computed using the Cartesian position fixes. A filter with 
highly accurate elevation may be more useful for studying geology or other terrain characteristics that are 
elevation-dependent. In general, the D-RATS data has precise elevation because it uses a digital elevation 
map as part of the solution computation. The elevation, d_0f is given as a distance vector from the center 
of the earth (imaginary center of the Cartesian plane) to the current position fix: 


do f '■= Uf 2 +y f 2 +z f 2 
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Filter speed is determined from output velocity vectors provided by the filtering algorithm. Typically 
filters will compute the velocity vector using position deltas, however this is not required. The current 
filter speed from velocity fixes, s _f, is given as: 

s f : = 

Filter speed can also be determined directly from the filter position fixes. In fact, since the sampling 
period is 1 sec for the Desert RATS data set, this quantity ( s '_fkfk- 1 ) is found in the same manner as the 
position delta, d _fkfk-\- 

s f(k)f(k- 1 ) : = 

The filter heading, determined from 1 -sec position fixes, is also a derivative quantity based on 
position fixes. The filter heading, hj k , can be combined with the filter speed s Jkfk-\ to provide an 
interesting look at the vector of the consecutive position fixes. In general, this vector should follow the 
test subject when the test subject’s location change exceeds the filter error. When the test subject stands 
still, the vector is less useful because the filter creates a scatter of position fixes around the true location; 
the vector simply maps the error. The corresponding equation for filter heading is given in degrees as: 




270 + tan 


180 + tan 


7 \ fol 

v a fk ~ la fk-i) 

( la fk ~ la fk- 1 ) 


l Cl j' k ^ Idj? k 


for la.f k > lcLf k and lOf k > lOf k 


for lcif k > lcLf k and lOf k < lc>f k 


for l-CLf k < l a f k 1 an d l°f k < t°f k t 


.1 0%~i ~ in / J 

( lo r k cos ( ia /J 


90 + tan 1 t r t 7 for la f < la f and lo f > lo f 

(, , \ f, ^ >k > k-i r k Ik- 1 


4.6 Ground Truth Solution Analysis 

Many of the same metrics can be applied to the ground truth measurements; however, the 
interpretation of these values is slightly different. For example, the position delta is a more useful metric 
since it is a better representation of the actual distance walked on a second-to-second basis. The ground 
truth data will contain some error, but the error is considered to be much less than that of the filter. 

The distance between two ground-truth position fixes (d_G k Gk~ i), the sum of those distances, and the 
elevation (d_0G) are calculated similar to the equations above: 



The ground truth speed (s_G) is calculated in a slightly different manner. The Novatel GPS receiver 
provides velocity along the X, Y, and Z Cartesian axes using instantaneous Doppler measurements, with a 
small latency of approximately 0.15 sec. Therefore, the speed computed using the receiver measurements 
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will not correspond identically to the filter speed, which is computed using position deltas. The ground 
truth speed and digital compass data are both distinct from other testbed measurements, since they are 
instantaneous. The quantity s_G is calculated according to the equation: 

s G ■ = Jvx G 2 + vy G 2 + vz G 2 


4.7 Filter Error Analysis 

The overall performance of the navigation filter can be characterized in terms of its position accuracy. 
All of the other filter quantities discussed above are derived from the filter position, therefore it follows 
that a highly accurate filter position will lead to a highly accurate speed, heading, etc. The solution 
position error ( e _pos ) is computed directly from the filter position and ground truth position: 

e pos : = J(x f ~ x G ) 2 + (y f - y G ) 2 + {z f - z G f 

The positive elevation error (e_elev) of the filter position estimate is computed using the imaginary 
vector quantities discussed above, originating at the center of the Cartesian axis. 

e elev ' ~ | d 0 f ~ d$ G | 

The positive speed error ( e speed) of the filter speed estimate is determined from the filter velocity 
estimates and the ground truth speed. It is important to keep in mind that the ground truth speed is 
determined using instantaneous Doppler measurements with slight delay. Ideally, the filter also should 
deliver instantaneous speed updates to the user. 

e speed 1 | s / — S G | 

The position delta speed error ( e speed _posdelta) compares the filter speed estimate determined 
using the position deltas between subsequent fixes and the instantaneous ground truth speed. When the 
test subject is walking at a relatively constant rate, these quantities should be very close. It is also possible 
to compare the filter position delta with the ground truth position delta, however that is not done here 
because the instantaneous Doppler velocities would be more accurate at the time when the speed value is 
updated. 


e speed_posdelta • — | s /(fc)/(k-l) S G 


4.8 Velocity Made Good 

One useful Measure of Effectiveness (MoE) is the velocity made good ( VMG ). VMG is the effective 
velocity of the test subject along the ideal path from the current ground truth location to the next 
waypoint. If the test subject is on the proper course, the magnitude of the VMG will be equal to the 
magnitude of the test subject’s velocity. If not, the VMG will always be smaller than the velocity. 
Effectiveness can be determined as the ratio of VMG to velocity. 

The equation for VMG is given as: 

_ On - x G ) * vx G + (y n - y G ) * vy G + (z n - z G ) * vz G 
V On - X G ) 2 + (y n - y G ) 2 + On “ Z G ) 2 
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5.0 Summary of Data Analysis Metrics 

Table II contains a summary of the metrics discussed in this paper with some commentary on 
application. 


TABLE II.— DATA ANALYSIS METRICS AND DISCUSSION 


Metric 

Description/commentary 

h _pn 

Ideal traverse heading function; provides the ideal heading from one waypoint to the next. 

• Used to determine the most efficient 2-D heading between waypoints. 

d _pn 

Ideal traverse distance function; provides the ideal distance from one waypoint to the next. 

• Used to determine the most efficient 2-D route between waypoints. 

h _pG 

Heading from the previous waypoint location to the current ground truth (GT) position. 

• Used as part of the 90° path deviance calculation; assumes the test subject started at the waypoint. This metric 
will confonn to the ideal heading ( h _pn) as the test subject approaches the next waypoint. 

d_PG 

Distance from the GT position closest to the previous waypoint to the current GT position. 

• Provides the distance baseline for the 90° path deviance calculation. 

deviance 

90° path deviation. 

• A very useful metric for examining path conformance. However, the metric assumes that the navigation 
system is guiding the test subject back to the ideal path between waypoints. 

hPG 

Heading between the final GT position of the previous segment and the current GT position. 

• More useful than h jG when the test subject does not begin the segment at the exact waypoint location. 

h Gn 

Heading from the current GT location to the next waypoint. 

• Useful when the navigation system does not guide the test subject back to the ideal path. This heading 
represents the shortest line between the current location and the destination waypoint. 

h_D 

Heading determined from the GT digital compass. 

• Only used for ground analysis. Often this metric is not valid as a comparison since it records the direction the 
test subject is pointed and not necessary the direction of motion. 

h_G 

Heading detennined using the change in GT position over a 1-sec interval. 

• Best comparison to h _f when h J is derived directly from position, since it uses the same computation method. 

hj 

Heading detennined using the change in filter position over a 1-sec interval. 

• This is the value shown to the test subject, possibly with some smoothing or averaging depending on the filter. 

a _pGpn 

Angle between the headings from previous waypoint to current GT position and ideal waypoint-to-waypoint. 

• The ‘pG’ quantity is useful for seeing how well test subjects were able to navigate away from the starting 
waypoint. 

• However, ‘p n ’ only applies as long as the test subject stays on the ideal traverse path. 

a PGpn 

Angle between the headings from previous waypoint GT location to current GT position and ideal waypoint-to- 
waypoint. 

• ‘PG’ works well when the test subject does not start on top of the waypoint. 

• However, ‘pn’ is not a good comparison since the test subject does not start at point ‘p’. 

a Dpn 

Angle between digital compass heading and the ideal waypoint-to-waypoint heading. 

• ‘D’ is a true heading indicator, unlike all of the GPS-based course indicators. 

• ‘D’ indicates which direction the test subject is pointing, not which direction they are moving. 

• ‘pn’ baseline only applies as long as the test subject stays on the ideal line. 

• ‘D’ tends to have magnetic/electric errors that can confuse analysis of the data. 

a Jpn 

Angle between the heading shown to the test subject on the display and the ideal waypoint-to-waypoint heading. 

• f is the heading seen by the test subject on the display, so it is well-known to the test subject. 

• ‘pn’ baseline only applies as long as the test subject stays on the ideal line. 

a _fGn 

Angle between the displayed heading and the ideal heading from current GT location to the next waypoint. 

• ‘f is the heading seen by the test subject on the display, so it is well-known to the test subject. 

• ‘Gn’ baseline is fluid and will re-compute the ideal heading based on the current position. 

• However, ‘Gn’ baseline is not known by the test subject, since the ground truth location is not known. 
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TABLE II.— DATA ANALYSIS METRICS AND DISCUSSION 


Metric 

Description/commentary 

a JD 

Angle between the filter heading and the digital compass heading. 


• Provides useful analysis of deviation between filter course and instantaneous heading. 

• However, ‘f and ‘D’ come from different sensors and might not be 1-to-l comparable in all situations. 

a DGn 

Angle between the digital compass heading and the ideal heading to the next waypoint from current GT location. 


• Metric could be used to observe if the test subject is looking around (turning) for the waypoint. 

• It is generally not that useful to compare pointing direction with walking direction. 

a JG 

Angle between the filter heading and the ground truth heading using 1-sec position deltas. 


• ‘/’ could have smoothing that would be useful to compare to a metric without processing like ‘G’. 

• ‘G’ tends to have high levels of error when ground speed is low. 

a Gpn 

Angle between the ground truth 1-sec heading and the ideal waypoint-to-waypoint heading. 


• Similar to a Jpn, provides insight into actual walking direction versus ideal heading. 

• ‘pn’ baseline only applies as long as the test subject stays on the ideal line. 

dJ(k)f{k-\) 

Distance between consecutive filter position fixes. 


• Useful only as a qualitative indicator of filter position stability. 

d_0f 

Distance from the center of the Cartesian plane to the current filter position. 


• When compared to an elevation map, provides insight into the filter’s ability to tie solutions to the surface. 

• Should remain relatively constant, plus or minus small terrain variations, for surface navigation. 

S _f 

Speed determined by the filter. 


• Typically equivalent to d_f{k)flk- 1 ) unless the speed algorithm is separate from the position algorithm. 

d_G(k)G(k-\) 

Distance between consecutive GT position fixes. 


• Quite useful when summed for a given segment or test; provides a good estimate of total distance travelled. 

d_0G 

Distance from the center of the Cartesian plane to the current GT position. 


• Indicates the GPS’ view of the current elevation in WGS84; comparable to d Of. 

s_G 

Speed determined by the GPS unit. 


• Typically an instantaneous speed, only comparable to s _/’when walking speed is constant across the 
navigation filter update rate (1 sec here). 

e _pos 

Error, RSS, of the filter position versus the GT position. 


• Standard metric to indicate filter performance for a given position fix. 

e_elev 

Error, absolute value, of the filter elevation versus the GT elevation. 


• Indicates the filter’s ability to clamp the solution to the surface. 

• Also can indicate coordinate system mismatches between the GPS and navigation filter. 

e_speed 

Error, absolute value, of the filter speed estimate versus instantaneous GPS speed. 


• Not very useful unless the filter can also provide near-instantaneous speed estimates. 

• Typically this will provide better insight into filter position stability than speed. 

• Depending on implementation, similar to e speed _posdelta, which uses position fix deltas to generate speed. 

VMG 

Velocity made good, effective velocity of the test subject from the current GT position to the next waypoint. 


• Interesting method of analyzing the test subject’s progress in reaching a waypoint independent of heading. 

• Can be zero for perpendicular travel or negative for travel in the wrong direction. 


6.0 Recommendations 

The data analysis process provided a great deal of insight into the considerations needed for designing 
a useful test procedure. Data analysis touches every aspect of the data, from time alignment methods, to 
data formats, to the underlying research objectives and test methodology. 

The time alignment procedure for D-RATS 2009 was sufficient to reconstruct the dataset, however 
this needed to be done by hand and took a great deal of time. There were four primary timestamps used 
by LECLDS: a system timestamp (ns), the GPS time (ms), processor ticks (ticks/sec), and the handwritten 
test log timestamp (min). It was difficult to tie the handwritten test log timestamp to the other timestamps 
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since the lack of resolution and u nk nown bias was significant. It would be better if all of the logs shared a 
common timestamp with similar resolution; for example, record the test log using a GPS timestamp with 
accuracy to the nearest second. 

LECLDS used multiple log files for the different testbed sensor data. This was most convenient at the 
time, and allowed development of the software to occur independently. However, a single, integrated log 
file would have been easier to analyze. It would have also solved many of the time alignment issues. 

The waypoint locations often did not correspond with the walking path. It is possible that this was due 
to the use of a separate, handheld GPS receiver to compute the waypoint location. Future tests should use 
the LECLDS GPS receiver to compute the waypoint position for consistency. It complicated analysis 
when the test subject did not end at the position of the destination waypoint. 

The display software was written to take the test subject to the next waypoint using the most direct 
route. This makes sense from a logistics sense, however in a real lunar scenario it is possible that there 
would be a safety risk in using this approach. Future tests should consider taking the test subject back to 
the original traverse path between waypoints, as that might be the safest path to use to traverse the area. 
This would simplify analysis and make path conformance a stronger metric. 

A bit of hand-waving was done when removing a collection of position fixes recorded around the 
starting or ending waypoints. It would have made the analysis more accurate if the test subject had some 
method of indicating the time of arrival (to the nearest second) and departure at a waypoint. It might be 
interesting to conduct the test without physical waypoints; that is, there would be no “cones” or visual 
clues in the field to indicate arrival at a waypoint. The test subject would have to rely on the display to 
accomplish the test objectives. 

Walking speed was not controlled in the test, since different test subjects would walk at different 
speeds depending on the terrain. If the test subjects were instructed to walk as quickly as possible (e.g., 
targeting 10 km in 2 hr), then the walking speed might be a good indicator of the test subject’s level of 
comfort with the system accuracy. 

7.0 Conclusions 

The data analysis techniques presented in this paper represent only a small subset of all of the 
possible methods of analyzing the data. These techniques provided insight into redesigns of the LECLDS 
testbed and changes in the test procedures, with the intention of collecting more useful data in the future. 

The Lunar Surface Navigation team at GRC conducted additional testing at the Desert RATS 2010 
event in September 2010. The test incoiporated many of the recommended improvements, including 
uniform time alignment, strictly-defined waypoint locations, navigation back to a predefined ideal path, 
and removal of cones from the test site. As of this report, revised data analysis metrics and methodology 
have not yet been defined. 
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